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BEM’S “FEELING THE FUTURE” (2011) FIVE YEARS LATER: 
ITS IMPACT ON SCIENTIFIC LITERATURE 


By Bruno A. Silva and Gabrielle Poeschl 


ABSTRACT: The study analyses the impact on scientific literature of the controversial 2011 article by Bem, 
“Feeling the Future.” Texts that cite Bem's article (V= 162) were identified from the Elsevier Scopus database 
for the years 2011 to 2015. Their summaries were analyzed using the Iramuteq program for textual data. The 
analysis suggested that the impact can be grouped into four classes: (a) The Replication class is characterized 
by a vocabulary addressing the role of replication in psychology research; (b) the Bayesian class reflects the 
perceived merits of this approach when compared with more traditional inferential statistics, namely statistics 
relying on p values; (c) the Experimental Studies of Anomalous Experiences (AE) class includes terms related 
to classical empirical research, applied in AE studies, and concepts, methodologies and theories specific to 
parapsychological research; (d) the Quantum Phenomena and Theories class suggests that quantum theories of 
brain/consciousness function may leave the door open to the possibility of the existence of psi phenomena. The 
Replication and Bayesian classes are represented more in psychology literature, confirming our hypothesis that 
psychology sources have a more critical position. Data moreover suggest that replication is currently the most 
referred path in the attempt to reach closure on this controversy. 
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Even before its publication, in March 2011, in the highly influential Journal of Personality and Social 
Psychology (JPSP), Daryl Bem's (2011) article “Feeling the Future: Experimental Evidence for Anomalous 
Retroactive Influences on Cognition and Affect” had already initiated an intense debate in the (social) 
media, establishing a remarkable controversy (e.g., Gad-el-Hak, 2011; MSNBC, 2011). The main reason for 
this was the message conveyed by its findings: “Precognition is scientifically supported” (Duggan, 2015). 
Although there is great public interest in parapsychological (psi) research, controversy has always been 
an integral part of this field of study, such that Richard Broughton (1991) titled his book Parapsychology: 
The Controversial Science. Evidence for psi was reported in the extensive parapsychological literature 
of recent and past years. What, then, made the article by Bem so unique and interesting? According to 
Palmer (2015), its particular importance lies in the scientific status of Bem as a social psychologist at the 
prestigious Cornell University, where he works, and in the fact that in this article he adapted experiments 
from methodological paradigms that are common in psychology, although reversing the order in which the 
task stimuli were presented. 

Although controversies in science are common and scientists often argue among themselves, this 
does not mean that scientists must argue or that controversies are essential to the development of science. 
The history of science shows that some important theoretical changes have been controversial (e.g., 
Darwinism), whereas other achievements (e.g., electricity-magnetism unification) have not (Machamer, 
Pera, & Baltas, 2000). There are also controversies that are explicitly avoided. By publishing in a highly 
regarded journal, and using well established methodological paradigms of mainstream psychology that 
were reversed, Bem also gave rise to controversy within the scientific community. 

In parapsychology, some controversies have been documented and studied. Zingrone (2006) 
presents an impressive amount of data and an in-depth analysis of controversies throughout its history. 
Palmer (2015) analyzed Bem's (2011) controversy in some depth, namely the specific criticisms levelled 
at his studies and the meta-analysis conducted by Tressoldi, Rabeyron, Duggan, and Bem (2016; see also 
Bem, Tressoldi, Rabeyron, and Duggan, 2015), which supports the early findings of Bem. 

The present study aims to complement this more in-depth approach to the controversies in 
parapsychology through an analysis of texts indexed in a scientific database that cited Bem's (2011) 


34 The Journal of Parapsychology 


article, using the Iramuteq program for textual data analysis (Ratinaud, 2009). Using exploratory means, 
Iramuteq employs a quantitative approach that we used in order to understand the impact of this article on 
the scientific community. 

We analyzed three supplementary variables representing the year of publication and subject area. 
The choice of the supplementary variables resulted from our expectations. Indeed, we did not have any 
particular predictions regarding the year of publication, but we had some hypotheses with respect to the 
subject areas in which the sources were indexed: We expected that a more critical point of view on Bem’s 
article would be demonstrated by papers published in sources indexed in the subject area Psychology, and a 
more positive point of view would be found in texts published in sources indexed in other subject areas (see, 
e.g., McClenon, 1982; McClenon, Roig, Smith & Ferrier, 2003; McConnell & Clark, 1991; Roe, 2016). 


Method 
Data Collection 


The texts that cite Bem’s (2011) “Feeling the Future” were identified from the Elsevier Scopus 
database for the years 2011 to 2015. The export procedure was performed on August 24, 2015, resulting in 
the collection of 163 texts, but one that was duplicated was excluded from further analysis. The following 
information was registered for the 162 remaining texts: (a) title, (b) abstract (when available), (c) authors” 
keywords (when available), (d) year of publication, (e) authors’ names, (f) source, and (g) subject area. A 
source could be indexed in more than one subject area. Very few errors were detected in the downloaded 
texts (eight words misspelled, two authors? keywords missing and one incomplete). These errors were 
manually corrected after consulting the original texts, and one title was translated from Italian to English in 
order to run the automatic data analysis on a reliable document. Titles, abstracts, authors? keywords, years 
of publication, and authors” names were then compiled into a single textual corpus, which constituted the 
material to be analyzed. 

Summaries were separated by the supplementary variables necessary for the association of the 
lexical productions with the years and subject areas of publication. Three supplementary variables were 
added: first, the year of publication [variable with five modalities (or categories): from 2011 to 2015], and 
then two different (but complementary) variables for the definition of the subject areas (e.g., Psychology, 
Medicine) in which the sources (academic journals, books, conference proceedings) were indexed: (a) a 
variable with two modalities: in the subject area Psychology, indexation (1) or not (11), and (b) a variable 
with three modalities: indexation only in the subject area Psychology (1), or indexation in the subject area 
Psychology as well as in other subject areas, namely, Arts and Humanities, Medicine, Social Sciences, 
Mathematics, Neuroscience, Business, Management and Accounting, Decision Sciences, Economics, 
Econometrics and Finance (ii), or indexation not in subject area Psychology but in other subject areas, 
which were, in addition to the above-mentioned ones, Physics and Astronomy, Multidisciplinary, Nursing, 
Agricultural and Biological Sciences, Computer Science, Engineering, Materials Science, and Health 
Professions (iii). The introduction of this second variable for the subject areas in which the sources were 
indexed was meant to assess whether the lexical production of the texts included in sources indexed in 
subject area Psychology and also in other subject areas would modulate the broad difference expected 
between the texts indexed or not indexed in subject area Psychology. 


Data Analysis 


To analyze the very large corpus extracted from the database, we used the Iramuteq program 
(Interface de R pour les Analyses Multidimensionnelles de Textes et de Questionnaires). This program, 
anchored in R software, enables several different statistical analyses and processing of text (Ratinaud, 
2009). In particular, Iramuteq enables the analysis of a corpus using a method developed by Reinert (e.g., 
1993), named Alceste (Analyse des Lexémes Cooccurrents dans les Enoncés Simples d’un Texte), which 
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has been available for a long time in the software ALCESTE®. The method enables the study of the formal 
structure of the co-occurrence of the words (“the lexical worlds”) in a given text or set of texts: Classes 
of words are created and associated with supplementary variables likely to explain the conditions of their 
production. More precisely, the program performs a downward hierarchical classification, extracting 
(creating) classes of words that co-occur and that are most dissimilar from other classes of co-occurring 
words; it presents the resulting clustering tree (dendrogram) showing the proximity relationships between 
the classes; 1t also provides a list of the most representative words in each class, a list of text segments in 
which the representative words are included, and the modalities (1.e., the categories) of the supplementary 
variables associated with the classes. Iramuteq also performs an analysis of correspondence (Benzécri, 
1973), which provides a schematic spatial representation of the relationships between the extracted classes, 
represented by their most representative words. Data analysis was performed using Iramuteq version 07.2, 
released on December 23, 2014. 

Concretely, three steps are taken to perform the analyses. In Step 1 the different parts of a text or, in 
the case of a questionnaire, respondents” answers to an open-ended question (see, e.g., Poeschl, Valentim, 
& Silva, 2015), are divided into text segments. The text segments correspond, usually, to a sentence and 
they are the basis of all computed statistics. At that time, all words are registered to create a dictionary, as 
well as a dictionary of their lemmatized forms (words reduced to their roots), intended to avoid irrelevant 
differences (words written in singular or plural, verbs conjugated in different tenses, etc.; Chabchoub, 
2008). The text segments are congregated in larger groups of text segments which form the rows of a table. 
The lemmatized forms form the columns of the table. The intersection of the rows and the columns registers 
either 1 if the form is present in the text segments or 0 if it is not. 

In Step 2, the table is submitted to a downward hierarchical classification, which iteratively 
separates the text segments into classes in which specific lemmatized forms co-occur and classes in which 
they do not (Reinert, 1986). This procedure was developed to analyze large logical tables (codified 0 or 1) 
of scarce frequencies. The first iteration decomposes the original table, made up of all the text segments, 
into two subtables, one including text segments with mostly 0 in the cells in the original table, and the other 
including text segments with mostly | in the cells. Each of the subtables thus constitutes a class made up 
of a distinct vocabulary with the overlap of classes as small as possible. The following iterations further 
decompose the subclasses, increasing the intraclass homogeneity and the interclass differences until the 
iterations fails to improve the sorting (Reinert, 1986; for more details about the procedure, see also Reinert, 
1983, 1985). The procedure results in a tree diagram. 

In Step 3, the results that can be interpreted are displayed. The classes presented in the clustering 
tree are described by the list of their associated forms, which constitutes the classes’ profiles. The profile of 
a class is the list of the words associated with the class. The degree of association of a form with a class is 
assessed by applying the chi-square formula on the observed data. An illustration of the relation of Form 
1 (Replication) with Class 1 (The Role of Replication in Psychology Research) is presented in Table 1 
(Reinert, 1993; see Pélissier, 2016, for further examples). In this example, the chi square of association is 
significant: y°(1, N = 622) = 105.27, p < .0001. 


Table 1 
Presence Versus Absence of Form 1 in the Text Segments of Class 1 Versus 
Other Classes in the Observed Data 


Number of text segments Number of text segments Total 

with Form 1 present with Form 1 absent 
Class 1 55 140 195 
Other classes 7 420 427 


Total 62 560 622 
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To indicate the direction of the relation between a form and a class (in Table 1, e.g, whether there 
were, or not, more Form 1 present in Class 1 than in the other classes, or vice versa), the sign of the 
difference between the observed and the expected frequencies is added to the chi square value, which is 
then denominated chi square of association. The significance of the chi square value, with one degree of 
freedom, is assessed and the classes are described by their most representative forms. The larger the chi 
square value, the stronger the association of the form with the class, and each class is thus characterized 
by a set of forms (words) more representative of that class than the others (Viaud, Uribe, & Acosta, 2007). 

A chi-square of association is also computed to assess the degree of association of the supplementary 
variables introduced in the corpus with the classes extracted by the program. An illustration of the relation 
of Modality 2 (year 2012) of Supplementary Variable 1 (Year of Publication) with Class 2 (Experimental 
Studies of Anomalous Experiences) is presented in Table 2. The chi square of association is significant: 
y2(1, N = 622) = 22.34, p <.0001. 


Table 2 
Presence Versus Absence of Modality 2 of Supplementary Variable 1 in the Text Segments of 
Class 2 Versus Other Classes in the Observed Data 


Number of text segments Number of text segments Total 
with Modality 2 present with Modality 2 absent 


Class 2 63 90 153 
Other classes 102 367 469 
Total 165 457 622 


We present hereafter the results obtained from the procedure. More specifically, we describe the 
profiles of the classes extracted by the downward hierarchical classification performed on the corpus, and 
their association with the modalities of the supplementary variables. Then we present the results of the 
correspondence analysis. 


Results 


The analysis of the corpus with the Iramuteq program was performed on the 162 texts that cited 
Bem’s (2011) “Feeling the Future.” In step 1, the corpus was subdivided by the program into 721 text 
segments containing, altogether, 4438 lemmatized forms. In step 2, the downward hierarchical classification 
sorted 622 text segments into four classes, classifying 86.27% of the 721 total text segments. The excluded 
segments contained forms (1.e., words) that were too rare to be taken into consideration (1.e., total frequency 
less than 3). The analyzed text segments were first divided into two main branches of the clustering tree (or 
tree diagram) on the basis of the dissimilarity of their vocabulary. Each one was subsequently decomposed 
further into two classes of words that did not require any further division (see Figure 1). 


First Branch of the Dendrogram 


We begin by describing the two classes of the first branch of the dendrogram presented in Figure 1, 
which includes 58.3% of the total text segments: the large Class 1 and the smaller Class 4. Table 3 presents 
a summary of the forms included in these two classes, that is, their 35 most representative forms, showing 
the frequency of each form in the class, its total frequency in the analyzed segments, the percentage of the 
form in the class, and the value of its chi square of association, calculated from a table such as Table 1 (for 
each value, df= 1; N= 622, p < .001). To make the reading easier, we substituted for each lemmatized form 
the word most frequently used in the corpus. 
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Figure 1. Dendrogram and relative weights of the classes based on the percentage of text segments. 


Class 1 - The Role of Replication in Psychology Research. Class 1, one of the two largest 
classes, is composed of 195 text segments, that is, 31.4% of the corpus, organized around “replication”: 
X = 105.27. As may be seen in Table 3, it also includes significant chi squares for the words “psychology” 
(X = 88.38), “science” (4? = 76.15), “research” (x? = 72.64), “article” (x? = 37.84), “publication bias” 
(X = 31.99), “practice” (x? = 31.76), “psychological” (y? = 27.72), “important” (x? = 27.44), “method” 
(x? = 26.28), “empirical” (yx? = 25.96), “education” (yx? = 25.18), “field” (x? = 22.72), “publication” 
(x? = 20.68), “methodological” (y? = 19.67), “recommendation” (4? = 18.46), “improve” (x? = 16.90), 
“credibility” (y? = 15.50), “validity” (42 = 13.27), “crisis” (x? = 13.25), and “error” (yx? = 11.87). This 
class is characterized by a vocabulary that addresses the role of replication in psychology research. The 
lack of replication of some studies, along with questionable research practices, may have contributed 
to a crisis of credibility regarding psychological science. Publication bias is a major concern, and the 
perceived crisis is also viewed as an opportunity to improve research practices through implementation 
of recommendations that can increase the credibility of scientific evidence in psychology journals. 
Issues of validity of empirical findings and errors also characterize the vocabulary of this class. 

This first class groups vocabulary that was referred to mostly in texts whose sources are 
indexed in subject area Psychology (f= 147, y? = 36.96). Significant chi squares were found for only 
subject area Psychology (f= 68, x? = 13.44); Psychology and Other Subject Areas (f= 79, x? = 9.1); 
and texts published in 2015 (f= 20, y= 11.46). The other chi squares are nonsignificant. With regard 
to the subject areas, we note that, although globally the vocabulary stems from texts whose sources are 
indexed in subject area Psychology, it is more specifically characteristic of the texts whose sources are 
indexed only in this area than of the texts whose sources are indexed in the modality Psychology and 
also other subject areas. 
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Table 3 
Class profiles: 35 Most Representative Words Associated With Class 1 and Class 
Class 1 Class 4 
Form Nin Total % in x? Form Nin Total %in x 
class N class class N class 
Replication 55 62 88.71 105.27 Bayes 31 31 100.00 88.89 
Psychology 62 81 76.54 88.38 Factor 35 42 83.33 73.17 
Science 47 57 82.46 76.15 Bayesian 32 39 82.05 64.56 
Research 61 86 70.93 72.64 Test 41 61 67.21 56.11 
Scientific 31 40 77.50 42.30 Null 25 29 86.21 54.57 
Article 30 40 75.00 37.84 Bayes factor 18 19 94.74 45.99 
Publication bias 16 17 94.12 31.99 Statistical 27 36 75.00 45.11 
Practice 31 45 68.89 31.76 Hypothesis 35 54 64.81 43.40 
Psychological 24 33 72.73 27.72 Datum 42 71 59.15 42.59 
Important 14 15 93.33 27.44 p values 16 17 94.12 40.27 
Method 29 44 65.91 26.28 Inference 14 17 82.35 27.41 
Empirical 21 28 75.00 25.96 Default 9 9 100.00 24.88 
Education 13 14 92.86 25.18 Set 17 25 68.00 22.46 
Field 16 20 80.00 22.72 Analyse 13 17 76.47 21.91 
Publication 11 12 91.67 20.68 Provide 14 19 73.68 21.89 
Methodological 12 14 85.71 19.67 Significance 10 12 83.33 19.88 
Recommendation 10 11 90.91 18.46 Calorie y 7 100.00 19.29 
Core 8 8 100.00 17.75 Frequentist 7 7 100.00 19.29 
Fact 8 8 100.00 17.75 Food 7 7 100.00 19.29 
Improve 12 15 80.00 16.90 Analyze 8 9 88.89 17.90 
Movement 9 10 90.00 16.24 Effect 31 63 49.21 17.84 
Credibility 7 7 100.00 15.50 Obtain 9 11 81.82 1723 
Issue 12 16 75.00 14.54 Evidence 34 72 4722 17.21 
Replicability 11 14 78.57 14.84 Assessment 11 15 7333 16.91 
Journal 14 20 70.00 14.34 Effect size 11 15 7333 16,91 
Developmental 6 6 100.00 13.27 Probability 11 15 73.33 16.91 
Guide 6 6 100.00 13.27 Stop 6 6 100.00 16.51 
Validity 6 6 100.00 13.27 Vield 6 6 100.00 16.51 
Crisis 9 11 81.82 13.25 Estimate 8 10 80.00 14.62 
Investigation 9 11 81.82 13.25 Favor 8 10 80.00 14.62 
Argue 16 25 64.00 12.90 Confirmatory 5 5 100.00 13.73 
Error 7 8 87.50 11.87 Exploratory 5 5 100.00 13.73 
Similar 7 8 87.50 11.87 One-sided 5 5 100.00 13.73 
Finding 17 28 60.71 11.75 Small 5 5 100.00 13.73 
Experimental 26 49 53.06 11.65 Value 12 19 63.13 13.16 


Note. N = number. Critical values of y? with 1 df: 19.51 for p < .00001: 15.14 for p <.0001; 10.83 for p < .001. 
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Class 4—Bayesian Statistical Inference. Close to Class 1 in terms of vocabulary, Class 4, the 
second largest class, includes 167 text segments which correspond to 26.9% of the total segments. This class 
is structured around “Bayes” (x? = 88.89). As shown in Table 3, it also comprises significant chi squares 
for the words “factor” (y? = 73.17), “Bayesian” (y? = 64.56), “test” (y? = 56.11), “null” ( = 54.57), “Bayes 
factor” (y? = 45.99), “statistical” (y? = 45.11), “hypothesis” (x? = 43.40), “data” (y? = 42.59), “p values” 
(X = 40.27), “inference” (y? = 27.41), “default” (y? = 24.88), “set” (y? = 22.46), “analyse” (y? = 21.91), 
“significance” (yx? = 19.88), “frequentist” (x? = 19.29), “assessment” (x? = 16.91), “effect size” (y? = 16.91), 
“probability” (x = 16.91), “exploratory” (y? = 13.73), and “one-sided” (x? = 13.73). The vocabulary of this 
class points mainly to concepts related with the Bayesian approach. The segments of text mostly reflect the 
perceived merits of this approach when compared with the more traditional inferential statistics, namely 
those relying on p values. The words “Bayes” and “Bayesian” are used to express an inferential statistical 
method used to test hypotheses (null and alternative) in which the notion of probability is different from the 
frequentist approach. On the other hand, the specific term “Bayes factor” is a value that results from one 
Bayesian approach to assessing null values, and it is used to infer which model (null or alternative) is more 
accurate given the data. With regard to effect size, it relates to both approaches. 

Class 4 groups vocabulary that is referred to more in texts whose sources are indexed in subject 
area Psychology (f= 121, x? = 20.74); Psychology and Other Subject Areas (f= 78, yx? = 22.16); and Texts 
Published in 2011 (f= 51, x? = 14.39). 


Second Branch of the Dendrogram 


The second branch of the dendrogram presented in Figure | includes 41.8% of the text segments, 
also divided into two classes of unequal weight, Class 2 and the small Class 3. Table 4 presents a summary 
of the forms included in these two classes, that is, their 35 most representative forms, indicating for each 
form its frequency in the class, its total frequency in the analyzed text segments, its percentage in the 
class, and the value of its chi square of association (df= 1, N = 622, p < .001). We also substituted for each 
lemmatized form the word most frequently used in the corpus, to make the reading easier. 

Class 2—Experimental Studies of Anomalous Experience. Class 2 includes 153 text segments, 
that is, 24.6% of the segments. It is organized around “participant” and “task” (42 = 43.10). It significantly 
comprises the words “experience” (42 = 34.67),“target” (y2 = 31.15), “response” (42 = 29.14), “predict” 
02 = 28.44), “learn” (42 = 27.99), “anomalous” (42 = 26.55), “nonintentional” (42 = 24.84), “spontaneous” 
(2 = 24.84), “stimulus” (y2 = 22.40), “performance” (42 = 21.70), “precognition” (42 = 20.43), “subject” 
(2 = 19.68), “emotional” (42 = 18.57), “prestimulus” (42 = 18.57), “mind” (y2 = 18.19), “control” (%2 = 
17.69), “randomly” (y2 = 17.29), “go/nogo” (%2 = 15.45), “PMIR” (v2 = 15.45), and “remote viewing” (42 
= 15.45). This class includes terms from classical empirical research, applied in anomalous experiences 
studies, and also concepts, methodologies, and theories specific to parapsychological research (e.g., non- 
intentional precognition task, prestimulus, go/nogo task, remote viewing, and psi-mediated instrumental 
response—PMIR). 

Class 2 groups vocabulary that was referred to mostly in texts whose sources are not indexed in 
subject area Psychology (f= 88, y2= 18.87); and in Texts Published in 2012 (f= 63, 42 = 22.34). 

Class 3—Quantum Phenomena and Theories. Class 3, with 107 text segments which corre- 
spond to 17.20% of the total analyzed segments, is the smallest of the extracted classes. It is significantly 
associated with “quantum” (%2 = 143.46), “brain” (y2 = 73.98), “consciousness” (y2 = 57.21), “Orch Or” 
([Penrose—Hameroff theory of] orchestrated objective reduction; 42 = 39.01), “entanglement” (%2 = 37.81), 
“moment” (42 = 37.81), “system” (y2 = 34.52), “retrocausality” (42 = 34.08), “space time” (y2 = 34.08), 
“microtubule” (42 = 29.16), “nonlocality” (42 = 29.16), “superposition” (42 = 29.16), “reality” (42 = 28.12), 
“mechanic” (42 = 24.26), “paa” (predictive anticipatory activity; 42 = 24.26), “causality” (y2 = 23.53), 
“principle” (y2 = 23.33), “anticipatory” (42 = 19.38), “EKT” ([analysis of] Echeverria, Klinkhammer, and 
Thorne; y2 = 19.38), “paradox” (42 = 19.38), “backward” (y2 = 18.60), and “bilk” (y2 = 18.60). 

This last class grouped vocabulary that was mainly mentioned in texts whose sources are not in- 
dexed in subject area Psychology (f= 82, y2 = 61.84); and in Texts Published in 2011 (f= 34; y2 = 10.26). 
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Table 4 
Class Profiles: 35 Most Representative Words Associated With Class 2 and Class 3 
Class 2 Class 3 
Form Nin Total %in x? Form Nin Total %in X 
class N class class N class 
Participant 22 29 75.86 43.10 Quantum 32 35 91.43 143.46 
Task 22 29 75.86 43.10 Brain 15 15 100.00 73.98 
Experience 20 28 71.43 34.67 Consciousness 16 20 80.00 57.21 
Target 10 10 100.00 31.15 Conscious 12 15 80.00 42.56 
Response 18 26 69.23 29.14 Orch or 8 8 100.00 39.01 
Predict 12 14 85.71 28.84 Entanglement 9 10 90.00 37.81 
Learn 9 9 100.00 27.99 Moment 9 10 90.00 37.81 
Anomalous 10 11 90.91 26.55 System 14 22 63.64 34.52 
Person 14 19 73.68 25.46 Retrocausality 7 7 100.00 34.08 
Time 21 35 60.00 25.06 Space time 7 7 100.00 34.08 
Nonintentional 8 8 100.00 24.84 Microtubule 6 6 100.00 29.16 
Spontaneous 8 8 100.00 24.84 Nonlocality 6 6 100.00 29.16 
Investigate 10 12 83.33 22.76 Superposition 6 6 100.00 29.16 
Stimulus 17 27 62.96 22.40 Reality 7 8 87.50 28.12 
Performance 7 7 100.00 21.70 Imply 5 5 100.00 24.26 
Precognition 19 33 57.58 20.43 Macroscopic 5 5 100.00 24.26 
Subject 11 15 73.33 19.68 Mechanic 5 5 100.00 24.26 
Emotional 6 6 100.00 18.57 Paa 5 5 100.00 24.26 
Prestimulus 6 6 100.00 18.57 Universe 5 5 100.00 24.26 
Mind 13 20 65.00 18.19 Basic 6 7 85.71 23.33 
Control 12 18 66.67 17.69 Causality 7 9 77.78 23.53 
Feel 7 87.50 17.29 Framework 7 9 77.78 23.53 
Randomly 7 87.50 17.29 Fundamental 7 9 TITR 23.53 
Read 7 8 87.50 17.29 Principle 6 7 85.71 23.33 
Word 7 87.50 17.29 Paper 10 17 58.82 21.26 
Reward 8 10 80.00 16.82 Anticipatory 4 4 100.00 19.38 
Belief 13 21 61.90 16.31 Biology 4 4 100.00 19.38 
Human 12 19 63.16 15.71 Ekt + 4 100.00 19.38 
Go nogo 5 5 100.00 15.45 Paradox 4 4 100.00 19.38 
Hit 5 5 100.00 15.45 Nature 10 18 55.56 19.14 
Trust 5 5 100.00 15.45 Physical 6 8 75.00 19.01 
Twin 5 5 100.00 15.45 Backward 5 6 83.33 18.60 
Wander 5 5 100.00 15.45 Bilk 5 6 83.33 18.60 
PMIR 5 5 100.00 15.45 Usual 5 6 83.33 18.60 
Remote viewing 5 5 100.00 15.45 Causal 8 13 61.64 18.32 


Note. N = number. Critical values of y? with 1 df: 19.51 for p < .00001: 15.14 for p <.0001; 10.83 for p < .001. 
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Relationships Between Classes 


The relationships of proximity and opposition between the classes may also be observed in 
the graphic provided by the correspondence analysis performed by the Iramuteq program. As may 
be seen in Figure 2, the most representative words of the four classes are positioned on two axes: the 
horizontal axis explains 40.75% of the inertia (1.e., variance) and the vertical axis 30.49%. 
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Figure 2. Correspondence analysis 


Figure 2 confirms, on the horizontal axis, the strong contrast between Classes 1 and 4 (The Role 
of Replication in Psychology Research, Bayesian Statistical Inference), and Classes 2 and 3 (Experimental 
Studies of Anomalous Experiences, Quantum Phenomena and Theories). If we take also into consideration 
the position of the words on the vertical axis, we may further note that the texts on Replication are the 
most closely related, in terms of vocabulary, with the texts on Bayesian Statistical Inference, and most 
distant from the texts on Experimental Studies of Anomalous Experiences. On the other hand, the texts on 
Quantum Phenomena and Theories are the most closely related, in terms of vocabulary, with the texts on 
Experimental Studies of Anomalous Experiences and most distant from the texts on Bayesian Statistical 
Inference. 
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Discussion 


The analysis performed on the summaries of published texts with regard to Bem's controversy 
extracted four classes, which enabled us to identify four types of impact on the scientific literature on Bem's 
(2011) article “Feeling the Future.” If Classes 1 and 4 (The Role of Replication in Psychology Research and 
Bayesian Statistical Inference) might be considered as coming from a more critical point of view, the data 
also show that they are close to each other and both are related with the identification of methodological 
problems and questionable research practices, including proposals for solving the problems. Classes 2 and 
3 (Experimental Studies of Anomalous Experiences and Quantum Phenomena and Theories), on the other 
hand, do not share the same point of view, inasmuch as they are characterized by a specific vocabulary 
which indicates different approaches to Bem’s article. 

Data suggest that, subjacent to the Replication class, and to some extent also the Bayesian class, 
it is possible to consider the question of the authenticity or ontological reality of psi phenomena. Is there 
evidence or strong evidence that psi is real? The lack of replication of some studies (not only in psi research, 
but, for example, also in mainstream psychology and medicine), questionable research practices, and the 
debate around some statistical inference methods, fomented criticisms and negative reactions to Bem’s 
findings. The Bayesian vocabulary was present more in 2011, when the debate began—for instance, in the 
same issue of JPSP in which the article by Bem appeared, Wagenmakers, Wetzels, Borsboom, and van der 
Maas (2011) published a paper on Bayesian statistics as a comment on Bem. On the contrary, 2015 was 
the year in which the question of replication seemed to be most prominent. In that same year, the journal 
Science listed the transformations made by psychologists in their research efforts (1.e., replications of key 
studies; creation of new models of scholarly publication and peer review, such as preregistration) as one of 
the notable scientific breakthroughs of 2015 (Bohannon, 2015). 

The technical and complex replication topic, and subtopics such as meta-analysis, have been 
addressed by some parapsychologists over the years (e.g., Honorton & Ferrari, 1989; Watt & Kennedy, 
2015, 2016). On the other hand, controversies between the supporters of frequentist and Bayesian forms 
of interpreting probability have a long history in science, with the former seeing probability as a limiting 
ratio in a sequence of repeatable events, and the latter as a mental construct that represents uncertainty and 
which applies not directly to events but to our knowledge about them. As argued by Howey (2002), “...a 
choice between the two interpretations of probability is not forced by pure logic or the mathematics of the 
situation, but rather depends on the experiences and aims of the individuals involved and their views of the 
correct form of scientific inquiry” (p. i). 

The fact that the Replication and Bayesian classes were represented more in psychology texts 
confirms our hypothesis that psychology sources have a more critical position concerning Bem’s (2011) 
article. Firstly, it was published in one of the most influential journals in social psychology, using common 
methodological paradigms of mainstream psychology, which were reversed; secondly, it is well established 
that psychologists in general have had a strong critical position about the existence of psi phenomena; thus, 
it was expected that the most critical reaction would come from psychology sources (although the Journal 
of Parapsychology is also indexed in the subject area Psychology, the relatively small number of articles 
it published on this topic reduced its impact in the scientific literature indexed in the selected database). 
When the sources indexed in the subject area Psychology were divided into Only Subject Area Psychology 
and Psychology and Other Subject Areas, the Bayesian class was represented more in the latter, which is 
consistent with the usual technical argumentation, particularly in mathematical terms. 

Classes 2, Experimental Studies of Anomalous Experiences, and 3, Quantum Phenomena and 
Theories, underlie two other types of impact of Bem’s (2011) article. As pointed out before, the class 
Experimental Studies of Anomalous Experiences includes terms from classical empirical research, 
applied in anomalous experiences studies, and also concepts, methodologies, and theories specific to 
parapsychological research. The data suggest that, for the authors of these texts, the citation of Bem's 
article was applied in a context of process-oriented research on anomalous experiences (experimental work 
designed to find evidence about the characteristics of processes that underlie anomalous experiences), as 
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opposed to proof-oriented research, that is, research that aims to find evidence for the existence of psi. 
This class is represented more in 2012 and in texts whose sources are indexed in subject areas other than 
psychology, thus confirming the hypothesis that sources from other areas of knowledge might have a more 
positive position concerning Bem’s article. 

The Quantum Phenomena and Theories class gives the same result, being more represented in 
texts whose sources are indexed in other areas of knowledge. In this case, data indicate that quantum 
theories of brain/consciousness function may leave the door open to the possibility of the existence of 
psi phenomena. For instance, retrocausality and predictive anticipatory activity (PAA) can be addressed 
in terms of quantum phenomena. This class, like the Bayesian Statistical Inference class, is represented 
more in the beginning of the debate (1.e., 2011). Scientists with a favorable position integrated the evidence 
of retrocausality phenomena as deserving further explanation, whereas the scientists with a more critical 
position pointed to the necessity for a new method of statistical inference to demonstrate that Bem’s (2011) 
analysis had weaknesses and was not valid. The data trends in the present study suggest that the replication 
path is currently the most referred to in the attempt to reach closure in this controversy. 
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Abstracts in Other Languages 


French 


PL ARTICLE “PRESSENTIR LE FUTUR” DE BEM (2011) CINQ ANS PLUS TARD : SON IMPACT 
SUR LA LITTERATURE SCIENTIFIQUE 


RESUME : Cette étude analyse l’impact sur la littérature scientifique de l’article controversé de Bem (2011), 
“Pressentir le futur”. Les textes qui citent l’article de Bem (V= 162) furent identifiés par la base de données 
Scopus d’Elsevier pour les années 2011 à 2015. Leurs résumés furent analysés par le programme Iramuteq 
pour les données textuelles. L'analyse suggère que l’impact peut être regroupé en quatre classes : (a) La 
classe de Réplication est caractérisée par un vocabulaire tourné vers la question du rôle de la réplication 
dans la recherche en psychologie ; (b) La classe Bayésienne réflète les mérites apparents de cette approche 
par rapport aux statistiques inférentielles plus traditionnelles, c’est-à-dire les statistiques s’appuyant sur les 
valeurs p ; (c) la classe des Etudes expérimentales des expériences anomales (AE) inclue des termes relatifs 
à la recherche empirique classique, appliquée aux études d’AE, et les concepts, méthodologies et théories 
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propres à la recherche parapsychologie ; (d) la classe des Phénomènes et théoriques quantiques suggère 
que des théories quantiques du cerveau et de la conscience pourraient ouvrir la porte à la possibilité de 
Pexistence des phénomènes psi. Les classes de Réplication et Bayésienne sont les plus représentées dans la 
littérature psychologique, confirmant notre hypothèse que les sources psychologiques sont davantage dans 
une position critique. Les données suggèrent également que la réplication est la voie à laquelle le plus grand 
nombre se réfère dans une tentative pour clore la controverse. 


German 


BEM‘S “FUHLEN DER ZUKUNFT” FUNF JAHRE DANACH: IHR EINFLUSS AUF DIE WISSEN- 
SCHAFTLICHE LITERATUR 


ZUSAMMENFASSUNG: Die Studie untersucht den Einfluss des 2011 erschienenen umstrittenen Artikels 
von Bem: ,,Feeling the Future“ auf die wissenschaftliche Literatur. In der Elsevier Scopus-Datenbank von 
2011 bis 2015 wurden N = 162 Texte identifiziert, die Bems Artikel zitieren. Ihre Abstracts wurden unter 
Verwendung des Iramuteq-Programms fiir Textdaten analysiert. Die Analyse ergab, dass ihr Einfluss in 
vier Klassen eingeteilt werden kann: (a) In der Replikationsklasse dominiert ein Sprachgebrauch, der die 
Rolle der Replikation in der psychologischen Forschung beschreibt; (b) die Bayesklasse hebt die Vorteile 
dieses Zugangs hervor, wenn er mit den mehr herkómmlichen schlieBenden Statistiken verglichen wird, 
und dies besonders bei Statistiken, die auf p-Werten beruhen; (c) die Klasse der Experimentellen Studien 
zu Anomalen Erfahrungen (AE) umfasst Begriffe, die sich auf klassische empirische Forschungen bezie- 
hen, angewandt auf AE-Studien und Konzepte, Methodologien und Theorien, die charakteristisch fiir die 
parapsychologische Forschung sind; (d) die Klasse der Quantenphänomene und Theorien legt nahe, dass 
Quantentheorien tiber die Funktionsweise von Gehirn/Bewusstsein die Méglichkeit offenlassen, dass Psi- 
Phánomene tatsáchlich existieren. Die zur Replikation und zu Bayes gehórenden Klassen finden sich eher 
in der psychologischen Literatur und bestatigen unsere Hypothese, dass psychologische Quellen eine eher 
kritische Position einnehmen. Dariiberhinaus legen die Daten nahe, dass auf Replikation gegenwärtig am 
háufigsten Bezug genommen wird beim Versuch, die Kontroverse zu beenden. 


Spanish 


EL ARTICULO DE BEM “FEELING THE FUTURE” (2011) CINCO ANOS DESPUES: SU IMPACTO 
EN LA LITERATURA CIENTIFICA 


RESUMEN: Este estudio analiza el impacto en la literatura cientifica del controvertido articulo de Bem 
“Feeling the Future” de 2011. Los textos que citan el articulo de Bem (N = 162) fueron identificados en la 
base de datos de Elsevier Scopus para los años 2011 a 2015. Sus resúmenes fueron analizados utilizando 
el programa Iramuteq para datos textuales. El análisis sugiere que el impacto se puede agrupar en cuatro 
clases: a) La clase de Replicación se caracteriza por un vocabulario que trata del papel de la replicación en 
la investigación psicológica; b) la clase Bayesiana refleja los méritos percibidos de este enfoque cuando se 
compara con estadísticas inferenciales más tradicionales, es decir estadísticas basadas en los valores de p; c) 
la clase de Estudios Experimentales de Experiencias Anómalas (EA) incluye términos relacionados con la 
investigación empírica clásica, aplicada a estudios de EA, y conceptos, metodologías, y teorías específicas 
a la investigación parapsicológica; (d) la clase de Fenómenos y Teorías Cuánticos sugiere que las teorías 
cuánticas de la función cerebro/conciencia pueden dejar la puerta abierta a la posibilidad de la existencia de 
fenómenos psi. Las clases de Replicación y Bayesiana están representadas más en la literatura psicológica, 
confirmando nuestra hipótesis de que las fuentes en psicología tienen una posición más crítica. Los datos 
sugieren además que la replicación es actualmente la vía más referida para tratar de llegar a una conclusión 
en esta controversia. 


